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, Abstract. In this article we connect topics from convex and integral geometry with well 

' known topics in representation theory of semisimple Lie groups by showing that the Cos^ 

. and Sin'^-transforms on the Grassmann manifolds Grp(K) ~ SU(n + 1, IK)/S(U(p, K) x 

r I ' U(n + 1 — p, K)) are standard intertwining operators between certain generalized principal 

. series representations induced from a maximal parabolic subgroup Pp of SL(n + 1, K). The 

' index p indicates the dependence of the parabolic on p. The general results of Knapp 

1-^^ , and Stein and Vogan and Wallach then show that both transforms have meromorphic 

d ' extension to C and are invertible for generic A G C. Furthermore, known methods from 

. representation theory combined with a Selberg type integral allow us to determine the 

I— I' iiT-spectrum of those operators. 

^ ! Introduction 

in . 

' The Cos^ and Sin'^ transforms have been widely studied during the last few years. One of 

2 ■ the reasons is their connection to convex geometry and to some classical integral transforms, 

^ . like the Funk and Radon transform on the sphere and their generalizations to Grassmann 

^ I manifolds. We will not discuss the history of these transforms but refer to [211 |25l [261 [2H] 

>■ ■ and references therein for further information. 

■ For the sphere S", the Cos'^ transform is defined on L^(S"') by 

C\f){u)= [ \ix,u)\'-Pfix)daix) (0.1) 

where da is the normalized rotational invariant measure on S", (■, ■) stands for the usual 
inner product on M" and p = {n + l)/2. The name Cos^ transform comes from the fact 
that for non-zero x and y the quantity (x, y)/||x|| is the cosine of the angle between x 
and y. It is clear that C'^ is in fact an integral transform on L'^{B), where B := Gri(M) is 
the Grassmann manifold of lines in M"+^. 
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Denote by Gr„(R) the Grassmann manifold of n-dimensional subspaces of R"+^. The 
Sin'^-transform iS^ maps functions on B to functions on Gr„(]R) and is given by 

Note that Gr„(M) B so that one can also view as an integral transform on L'^{B). 

As we will see later, the factor p in the exponent of (10.11) is chosen so that C'^ and 
iS'*' coincide with will known integral transforms from representation theory, i.e., standard 
intertwining operators between certain principal series representations of SL(?t, + 1, M). 

We note that C^{f) exists for all / G ^^(S") as long as A G Cp := G C | Re (2) > p} 
and A h-> C^{f) is holomorphic on C°, the interior of Cp. The transform : C°°(S") — > 
C°°(S") extends meromorphically to C. One way to determine the singularities of this 
extension is to determine the SO(n + l)-spectrum of C^. The group K = SO(n + 1) acts 
naturally on L^(S") and, as a representation of K, the space L^(S'^) decomposes into a 
direct sum 

l2(s'^) y"' 

meN() 

where 

= {p|s" I p a homogeneous polynomial of degree m and Ap = 0} . 

Here A = df + ■ ■ ■ + S^+i denotes the Laplace operator on M*^"*"^. Likewise, by considering 
Cos'^ as a transform on LF'{B), we have the decomposition 

L\B)^K y^. 

me2l% 

As these are multiplicity-one decompositions and commutes with rotations, it follows 
from Schur's Lemma that for each m G 2No there exists a holomorphic function 77^ : Cp — )■ 
C such that 

C'^'lym = ?7m(A)id3;m . 

Information about 7]^ gives more detailed information about the meromorphic extension 
of as well as the image and the kernel of the Cos'^'-transform, as clearly both of those 
spaces are if-invariant. 

The functions 77m(A) can be for instance determined using Funk-Hecke type formula 
[25], and one gets for m G 2No 

. r(p) r(i(A-p + i))r(i(-A + p + m)) 

^'"^ ^ ^ ^ r(i/2) r(i(A + p + m))r(i(-A + p)) " 

Note that 

F (^(-A + p + m)) _ r 1 if m = 

r (i(-A + p)) ~ \2-"^/2(-A + p)(-A + p + 2) ■ ■ ■ (-A + p + m - 2) if m G 2N 

is a polynomial of degree m/2. 
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The first pole is at A = (n — l)/2 and the Funk transform 

= / fix) 

is then given, up to a constant that can be determined, by J^{f){uj) = cResA=(n-i)/2C'^(/)(a;). 
A similar discussion relates the Radon transform with the singularities of the Sin^-transform. 

All of these integral transforms have been generalized to the Grassmann manifolds over 
M, C and M. We refer to [21 [221 [261 [271 [3l] for more details and discussions. 

The integral transform in (10 .ip appeared independently in representation theory of 
SL(n + 1, K), where IK = M, C or H, around the same time as the Cos''^-transform. The real 
case was studied in [10], the complex case in pj, and finally the quaternionic case in [23]. In 
these articles it was shown that the integral (10. ip defines an intertwining operator between 
generalized principal series representations induced from a maximal parabolic subgroup in 
SL(n + 1,K). The i^-spectrum was determined. Among the applications was the compo- 
sition series, some embeddings of the complementary series and the study of the so-called 
canonical representations on rank-one Riemannian symmetric spaces of the noncompact 
type, [71IS1[S|- The connections to convex geometry and to the Funk and Radon transforms 
was neither discussed nor mentioned. 

Let K = M,C or H and let Grp(K) = SU(n + 1, K)/S(U(p, K) x U(g,K)), with q = 
n + 1 —p, be the Grasmmann manifold of p-dimensional subspaces of K""^^. Our main aim 
in this article is to show that the Cos'^ and Sin'^-transforms for Grp(]K) are always canonical 
intertwining operators between generalized principal series representations induced from 
maximal parabolic subgroups of SL(?t, + 1, K). We then use the spectrum generating method 
developed in [5] to determine the ii"-spectrum of the Cos'^ and Sin'^-transform. 

The article is organized as follows. In Section [1] we recall needed notations and fact on 
simple Lie groups and Lie algebras. In Section [2] we introduce the generalized spherical 
principal series representations. These are representations induced from a character xx of 
a parabolic subgroup P = MAN. The elements of the representation space can be viewed 
as sections of the G-homogeneous line bundle over B = G/P corresponding to xx- We 
recall the results by Knapp and Stein and Vogan and Wallach on the meromorphic con- 
tinuation of standard intertwining operators J(A) between spherical generalized principal 
series representations, [T5l \TQ\ [30] . The material of this section is presented in the general 
setting of semisimple Lie groups rather than for the special case considered later in this 
article. Indeed, the analytic properties of the intertwining operators which we discuss here 
possess a rather unified structure for the different semisimple Lie groups. On the other 
hand, we will associate a geometric meaning to these operators only for a specific class of 
semisimple Lie groups. It is therefore a natural question whether one can provide a geo- 
metric meaning to the standard intertwining operators for other groups or, more generally, 
to intertwining operators between generalized principal series representations induced from 
other parabolic subgroups. We expect that the answer will be positive and that higher 
rank Cos'^-transforms are in fact given by such intertwining operators. 
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In Section [3] we recall the method from [5J on how to find the eigenvalues of the inter- 
twining operators on each of the i^'-types in case P is a maximal parabolic subgroup and 
the i^'-types have multiplicity one. As the situation considered in this article is less general 
then the one in [5], the results and their proofs are simpler. The principal result is Theorem 
13. 5[ which provides us with an inductive way to compute the spectrum. 

We specialize to the case of the Grassmann manifolds Grp(K) in Section HI We introduce 
the Cos^-transform and show that = J{X)- This main result has several applications. 
Using the well-known fact that the generalized principal series are irreducible for almost 
all A G C, we obtain that there exists a meromorphic function cp(A) = J(A)1 = C^{1) such 
that C^^ o = o C^^ = cp(— A)cp(A)id for almost all A. Moreover, together with the 
method from |5j outhned in Section [31 the equality = J(A) allows us to calculate the 
-ft"-spectrum of C'^. This computation is done in Section O The main result is Theorem 
15.101 Finally, a similar discussion about the Sin'^-transform is then carried out in Theorem 

m 

1. Semisimple Lie Groups and Parabolic Subgroups 

In this section we introduce the notation and basic facts on semisimple Lie groups. Lie 
algebras and parabolic subgroups. The material can be found in any standard textbook on 
semisimple Lie group, e.g. fri|. 

From now on G will stand for a noncompact connected semisimple Lie group with finite 
center. The Lie algebra of G is denoted by g. In general Lie groups will be denoted by 
upper case Latin letters and their Lie algebras by the corresponding German letter. 

An involution ^ : G — >■ G is a Cartan involution if K = G^ = {a & G \ 6{a) = a} is a 
maximal compact subgroup of G. In that case K is connected. From now on 6 will denote 
a fixed Cartan involution on G and K the corresponding maximal compact subgroup. Our 
standard example will be G = SL(n + 1,]K), K = M, C, or H, and 6{x) = (x*)^^ where 
denotes the canonical conjugation in K which fixes M. In this case K = {x ^ G \ = x}. 
The general structure theory discussed in this section is specialized to this case in Section 

SI 

The derived involution 9 : q g and its complex linear extension to gc = ®]r C will 
for simplicity also be denoted by 6. Then g = t © s where t = is the Lie algebra of K 
and s = {X G g I 6{X) = -X}. We have [s,s] C [!,s] C s, and Ad(i^)s = s. We denote 
the representation of K on Sc by Ad^. 

Denote by ( ■ , ■ ) the Killing form on g, given by (X, Y) = Tr(ad(X)ad(y)). Then 

{X,Y)^{X,Y)e := {X,9{Y)) 

is an inner product on g. If X G g then ad(X)* = — ad(^(X)). In particular, if X G 5 
then ad(X) is a symmetric operator and hence diagonalizable over the reals. Let a C s be 
abelian. For a G a* let 



g„ = {X G g I (Vi/ G a) [H, X] = a{H)X} 
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and let 

m = {X e q \ {yH e a) [H, X] = and X ± a} . 

Then m ® a = 0o- We will always assume that the 

a = 3(m) n s = {X e m e a I [X, tn e a] = {0}} . 

Let A = A{q, a) :— {a e a* \ {0} | Qa 7^ {0}}. The elements in A are called roots. We 
have 

As [Qa: Qp] ^ 0a+/3 for a, /3 G A U {0} it follows that [m © a, Qa] ^ 0a for all a e A. 

Let Or -.^ {H e a \ (Va e A) 7^ 0}. For a fixed e a,, let A+ := {a e A | 
a{H) > 0}. Then A+ is system of positive roots, i.e., A = A+ U -A+, A+ n -A+ = 0, 
and (A+ + A+) n A C A+. In particular, 

n := 0a 

is a nilpotent subalgebra of Q normalized by the Lie algebra p := m © a ffi n. In fact 

P = tXg(n) = {Xe0|[X,n]Cn}. 

p is a parabolic subalgebra of q. It is maximal, if dim a = 1. 

Let P = Ng{p) = {g E G \ Ad{g)p C p}. Then P is a closed subgroup of G with Lie 
algebra p. Let A = expa respectively N = expn denote the analytic subgroup of G with 
Lie algebra a respectively n. Then A and N are closed, A is abehan and is nilpotent. 
Let Mo denote the analytic subgroup of G with Lie algebra m and M :— Zk{A)Mo. Then 
M is a closed subgroup of G with finitely many connected components, Zg{A) — MA and 
the product map M x A x N ^ P, (m, a, n) 1-^ man, is an analytic diffeomorphism. Let 
L = KnM &ndB:= K/L. Then 

G = KP and KnP = L. (1.1) 

Thus B = G/P and G acts on B. To describe this action, write g = kp{g)mp{g)ap{g)np{g) 
with {kp{g),mp{g), ap{g),np{g)) E K x M x Ax N. If the role of P is clear, then we leave 
out the index p. The map G 3 g ^ (0(5'), ^((7)) E Ax N is analytic and g (-> a{g) is right 
MA'"-invariant. We can therefore view a( • ) as a map G/MN — >■ A. The elements k{g) and 
m(g') are not uniquely defined. However, the maps 

g 1-^ k{g)L e B and g 1-^ Lm{g) e L\M 

are well defined and analj^ic. Let bg — eL. Then we can describe the G action on B by 
g ■ {k ■ ho) = k{gk) - bo- If b = k ■ bo E B and g E G, then we set a{gb) :— a{gk). 

Let N := 6{N). The Lie algebra of iV is n = ^(n) = 0^e-A+ S"- Furthermore, q = 
n©m©a©n = n©p, and the map NxMxAxN^G, {n,m,a,n) nman, is an 
analytic diffeomorphism onto an open and dense subset NP of G of full measure. 
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Let P = MAN = 9{P). Then as a X-manifold B = G/P but the G-action in general is 
different. Denote this action for the moment by ~. 

Lemma 1.1. Let g E G and b E B. Then ap{gb) — ap{9{g)h)~^ , kp{gb) — kp{9{g)b) and 
g^b = e{g) ■ b. 

Proof. Let k E K he such that b — k ■ bo- Then g~b — kp{gk)L. We have 

gk = eie{g)k) 

= 9{kp{9{g)k)mp{9{g)k)ap{9{g)k)np{9{g)k)) 

= kp{9{g)k)9{mp{9{g)k))ap{9{g)k)-'9{np{9{g)k)) . 

The Lemma now follows as M is ^-stable and 9{np{9{g)k)) G iV. □ 

We note for later use that the the above argument shows that 

ap(gk) ^ ap(9(g)k)-K (1.2) 

We normalize the invariant measure on B and compact groups so that the total measure 
is one. If / e C{B) then 

/ f{b)db= [ f{k-bo)dk. 

Jb Jk 
For A e define a homomorphism x;^ : P ^ C by 

Xxim exp Hn) = e'^^''^^ , m e M, H e a, and ne N . 

We will also use the shorthand notation for X\{p)- For a e A let nia ■— dim Qa and let 
Pp := I J2aeA+ "^a<^ G a*. As P is fixed most of the time we simply write p instead of pp. 
We have = |detAd(p)|„|. 

Normalize the Haar measure dn on N such that 



[_a{n)-'^P dn ^ 1 



and let dn = 9{dn). We normalize the invariant measure on A respectively o such that the 
inversion formula for the abelian Fourier transform holds without additional constants, i.e., 
if / e C^{A) and /(A) = f^f{a)a-^da, then /(a) = J^, f{iX)a'^dX. 

Lemma 1.2. Let the notation be as above. 

(1) We can normalize the Haar measure on G and M such that for all f e L}{G) 

n.)i. = f f f f mman)a"dMmdadn 

Jk Jm J a Jn 

— f{nman)a^'^ dndmdadn . 

Jw Jm J a Jn 
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(2) The set N ■ bo G B is open, dense and of full measure. If f E then 

f{h)dh= f_f{n-ho)a{n)'^Pdn. (1.3) 

(3) Let f e L\B) andxeG. Then 

f{x-h)a{xh)-'^P dh= [ f{h)dh. (1.4) 

■ JB 

Proof. See [H], pp. 137-140 and p. 170, or [I9], §7. □ 

2. Generalized Principal Series Representations 

We now introduce the spherical generahzed principal series representations and their in- 
tertwining operators. We refer to [151 EHl EQ] for proofs. Later we will show that those 
intertwining operators coincide with the Cos^-transform. 

For A G define a continuous representation of G on L'^{B) by 

[7r,(x)/](6) := a{x-^h)->^-Pf{x-' ■ h) . (2.1) 

Then vr^ = np^x = IndpXA- It follows easily by (11. 4p that vr^ is unitary if and only if A G ia* . 

Theorem 2.1 ( Vogan- Wallach) . There exists a countable collection {pn\n of non-zero holo- 
morphic polynomials on such that A G and pn{X) 7^ 0, then 7T\ is irreducible. In 
particular, tt\ is irreducible for almost all A G a^- 

Proof. This is Lemma 5.3 in [30]. □ 

Replace P by P, realize B as G/P and then induce the character determined by A from 
P to G. We denote the corresponding representation by n-p y 

Lemma 2.2. Let tt^ := ttpx o $. Then tt^ ^ ~ ^-a '^■^ ^ representation of G acting on 
L\B). 

Proof. This follows easily from Lemma II. H equation (11.21) and pp = —p p : 

[^P,x{x)f]{b) = ap{x-'b)-'-^f{x-'^b) 

= ap{e{x)-^bY-fp f{e{x)-^ -b) 

= [7rp,_A(^(x))/](6). □ 
For / G C°^{B) and b = k ■ bo define the standard intertwining operator J(A) = Jp|p(A) 

by 

J(A)/(6) := [ a{n)-^-Pf{kn ■ bo) dn (2.2) 



N 
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whenever the integral exists. We note that this is well defined. Indeed, if m G L then 
a{mnm~^) = a{n), and the Haar measure on N is invariant under conjugation by m. Let 
b = k ■ bo = ki ■ bo- Then k = kim for some m & L. Thus 

a{n)^^~'^ f{kn ■ bo) dn = a(n)~'^~'' f (kiuin ■ bo) dn 

N J~N 

a(mnm^^)^^^^^ f {ki(mnm^^) ■ bo) dn 

_a{n)-^-Pf{kin-bo) dn . 
TV 

Recall that the space of smooth vectors for vr^ is C°°{B). In particular, it is independent 
of A. If X e then we define as usually 7r^(X) : C^^B) C°°(i3) by 

7r,(exp(tX))/-/ d 



N 



lim 

t^o t dt 



7r,(exp(tX))/ 



t=o 

where the derivative is taken in L'^{B). Then vr^ is a representation of g in C°°{B) and 
extends to a representation of the universal enveloping algebra U{g). If -u G U{q) then 
TT'^iu) : C^IB) -> C°°{B) is a differential operator. If A G write X = Xr + iXi with 
Xr,Xi G a*. 

Theorem 2.3 ( Vogan-Wallach) . The following holds: 

(1) There exists a constant ap such that if 

X G ac(ap) := {/i G | (V« G A+) {fiR, a) > ap} 

then the integral ^2.2^ converges absolutely. Furthermore, there exists a constant 
C > such that for all X G a£(ap) and f G C°^{B) 

||J(A)/||oo<C||/|U. (2.3) 

(2) /// G C°^(B) then 

o£(ap) 9 A ^ J(A)/ G C~(i3) 

is continuous and holomorphic on the interior {/i G | (Va G A+) {fiR,a) > ap}. 

(3) There exists polynomial functions 6 : — )■ C and : — ?■ U{q)^ such that if 
X G a* (ap) and f G C°°{B), then 

6(A)J(A)/ = J(A + 4p)7rA+4p(I^(A))/. 

In particular X h-j- J{X)f extends to a meromorphic function on a^. 

(4) The operator J{X) intertwines tta and tt^x- Thus, if x & G and f G C°°{B), then 

J(A)(7r,(x)/) = A(x)J(A)/. (2.4) 

Proof. Referring to [30], we have that (1) is Lemma 1.2, (2) is Lemma 1.3, (3) is Theorem 
1.5, and (4) is Lemma 1.2 for A G a^(ap). It extends to by meromorphic continuation. 

□ 
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Our situation is simpler than the general case considered in [30], therefore-and for 
completeness-we describe the main idea behind the proof. Let 



ap := max {a, p) > . 



(2.5) 



According to Lemma A. 3. 3 in 
A G aUap) 



we have (loga(?7,), a) > for all a G A^. Thus for 



lain 



and n ^ a{n) is integrable. Thus, the integral defining J{\) exists and A i— ?■ J{X)f is 
holomorphic and also (12.31) clearly follows. Note that (12.31) says in particular that J{X)f G 

Part (2) follows by the above estimates (12.31) that show that the integral defining J{X)f 
converges uniformly on any domain of the form a^(a) with a > ap. 

The proof of (3) uses tensoring with finite dimensional representations of G. Another 
way of showing that A t— t- J{X)f extends to a meromorphic function on is to use the last 
part of (12. 2p and ideas from [6l [191 120] using the Bernstein polynomial for n k-)- a{n)~^~^ 
and equation (12. 2p . 

We will show that J(A) is an intertwining operator after having proved Lemma [231 below. 

Lemma 2.4. Let ap be as in Ii2.5\) and let A G a^{ap). Let 1 < p < oo and let f G L'^{B). 
Then the integral h2.2^) exists and \J{\)f{k)\ < \\f\\p. In particular, J{X)f G Lp{B) and 

\\JWf\\v< 



Proof. The case p = oo is (12.31) . We can therefore assume that p < oo. Let q be such that 
i + i = 1. Then 2p = + ^p. As \a{n)-^-P\ < a{nY^f and Jj^ a{n)-'^P dn = 1, it follows 
that 



a{n)-^-Pf{kk{n) ■ bo) dn 



N 



< 



< 



a{n)-^P\f{kk{n) ■bo)\dn 



N 



N 



a{n) p^f{kk{n) ■ bo 



dn 



\f{kk{n) ■ bo)fa{n)~^P dn 



N 



1/p 



1/p 



umi'db 



1/p 



p ■ 



Here we used (II. 3p to transfer the integral over to an integral over B. 
For X G NP write 

X = np{x)mp{x)ap{x)np{x) 



□ 
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with np{x) G A^, mp{x) G M, ap{x) G A, and np{x) G A^. Note that x ap{x) is right 
MA^-invariant. We leave out the subscript p if it is clear which parabolic subgroup we are 
using. 

Lemma 2.5. Assume that X G a*^{ap). Let f G C°°{B). Then forkeK 

J(\)f(k)= [ ap{k-^hf-Pf{h)dh= [ ap{k'H)^-Pf{h)dh. 
Jk Jb 

In particular, J{\) is a convolution operator on L'^{B) if X E a^(ap). 

Proof. Let n eN. Then 

n = k{n)rn{n)a{n)n{n) 

= n{k{n))rnj^{k{n))a{k{n))njj{k{n))m 
= n{k{n))m'a{k{n))a{n)n' 

for some m' E M and n' G A^. Thus a{k{n)) = a{n)~^. Assume that A G a'^i^ap). Using 
f lL3|) we get 

f_a{n)-^-pf{kk{n) ■ K) dn = !_a{k{n)Y-P f{kk{n) ■ ho)a{n)-^P dn 

J N J N 

= I a{hf~Pf{kh)dh 
Jb 

= [ a{k-^hY-Pf{h)dh 
Jb 

as the measure on B is i^-invariant. □ 

We now show that J(A)7rA(x) = 'n-\{9{x))J{\). For this, let x G G and A G a}-{ap). 
Then 

J{\){'Kx{x)f){k)= [ a{k-^hf~Pa{x~^h)-^-Pf{k{x-^h))dh. (2.6) 
Jk 

We have 

a{k~^h) = a{k~^xx^^h) = a{k^^xk{x^^h))a{x^^h) . 
Hence we can rewrite fl2.6l) as 

J{\){'Kx{x)f){k) = [ a{k~^xk{x-^h)f-Pf{k{x-^h))a{x-^h)-^Pdh 

Jk 

= [ a{k-^xhf-''f{h)dh 
Jk 

= I a{{x-^k)-^h)^-Pf{h)dh. (2.7) 
Jk 

Now use Lemma 11.11 to write 

x~^k = kp{x^^ k)ma-p(x^^ k)n = kp{6{x^^)k)map{6{x~^)k)~^n 
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for some m & M and n & N and insert this into ( 12. 7p to get 

J{\){7rxix)f)ik) = a{e{x)-^k)^-f [ a{k{9{x)-^k)-^h)^-Pf{h)dh = n_x{0ix))J{\)fik). 

Jk 

As J{X) is i^'-intertwining and the only fT- invariant functions in L'^{B) are multiple 
of the constant function l(x) = 1, it follows that there exists a meromorphic function 
cp : — )• C such that J(A)1 = C(A). For A G ac('^-p) function cp(A) is given by the 
converging integral 

cp(A) := J(A)1 = f_a{n)-^-P dn = [ a{kY'P dk . (2.8) 
Jn Jk 

Theorem 2.6. Let the notation he as above. Then, as an identity of meromorphic func- 
tions, 

J(-A) J(A) = J(A) J(-A) = cp(A)cp(-A) id 
Furthermore, if \ E ia* , then the normalized operator cp{X)^^J{X) is unitary. 

Proof. We have 

Ji-\)Ji\)nxix) = Ji-\)n^xi0ix))J{\) = 7r,(a;) J(-A) J(A) 

as 6'^ = id. As tt\ is irreducible for A in an open dense subset of it follows that the 
J(— A)J(A) is a scalar multiple of the identity on that set. As J(— A)J(A)1 = cp(— A)cp(A) 
it follows that J(— A) J(A) = c(— A)c(A)id as a meromorphic function. The same argument 
shows that J(A) J(— A) = c(— A)c(A)id and (1) follows. The unitarity of cp(A)~^^(A) for A 
purely imaginary follows from [cp(A)~^ J(A)]* = cp(— A)^^ J(— A) as in this case A = —A. □ 

3. The Spectrum Generating Operator 

We will now specialize the situation to the case where G is simple and P is a maximal 
parabolic subgroup. Thus dim a = 1. Based on our application later on, we will also assume 
that A = {a, —a}. Hence n is abelian. Finally, we will assume that K is semisimple and 
that {K, L) is a symmetric pair corresponding to an involution t : K ^ K. This implies 
that each irreducible representation of K that occurs in L'^{B) has multiplicity one. 

Denote by K the set of equivalence classes of irreducible representations of K. For 
simplicity we will often write 5 E K where 5 is an irreducible representation of K., i.e., we 
identify the representation with its equivalence class. Denote by Vs the finite dimensional 
Hilbert space on which 5 acts and let d{5) := dim 14. Let 

■= {ueV5\ {\/k G L) 6{k)u = u} . 

Then, as K/L is a symmetric space, either dim V/" = or dim V/' = 1. We say that {S, Vs) 
is spherical if V/' 7^ {0}. Let Kl denote the subset of K of spherical representations. To 
describe Kl write t = [ © q where q = {X G t \ t{X) = —X}. Let b be a maximal abelian 
subspace of q. Denote by A{ the set of roots of be in tc- Note that C ib*. Let A^*" 
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be a set of positive roots and let pt be the corresponding half-sum of roots, counted with 
multiplicities. Denote by the Weyl group of A{. Finally we let 

A+ := {/X e tb* I (Va G A+) G Mq} . 

[a, a) 

Then the map tt h- (highest weight of tt) defines an injective map of Kl into A"*". This 
map is bijective if and only if B is simply connected. In general is isomorphic to a 
sublattice A~^{B) of A"*". We refer to p. 535, and the discussion in [21j, in particular 
the appendix, for more details. Let fi e A"'"(i3) and denote by vr^ the corresponding spherical 
representation. Let be the space on which vr^ acts and let (■, the inner product on 
making vr^ unitary. We also fix a unit vector G V^. Define : — )■ L'^iB) by 

Then is an isometric if-intertwining operator between vr^ and the left regular represen- 
tation L of on L'^{B). Let L'j^^iB) := Im(T^). Then L is an irreducible ii'-representation 
on L'^{B) and 

Lemma 3.1. For ^ G A^{B) let J^{\) := J(A)|i^2(g). There exists a meromorphic function 
r/^ : — > C such that 

(1) J^(A) =r7^(A)idL2(B). 

(2) r/o(A) = cp(A), 

(3) r/^(A)r/^(-A)=cp(A)cp(-A), 

Proof. J(A) is a G-intertwining operator between tt\ and vr^^^ by Theorem 12.31 (4). As 
/^-representations, vr^ and tt^^ agree with L. Hence J(A) is a i^-intertwining operator of 
L with itself. As each L^^{B) is irreducible and the multiplicity of vr^ in L^{B) is one, it 
follows by Schur's Lemma that we can define ?7/i(A) by (1). The claim now follows from our 
normalization J(A)1 = cp(A) and Theorem 12. 6[ □ 

We call {v^iX) | G A+(i3)} the /sT-spectrum of J(A). 

Our aim is to determine the functions rj^ using the results from [5j. Fix Ho & a such 
that a{Ho) = 1. Fix a G- invariant M-bilinear form ( ■ , ■ ) on g such that {Ho, Ho) = 1. For 

G a we have a{H) = {H, Ho). Denote the complex bilinear extension to 0c by the same 
symbol. Define a map u : Sc ^ C°°{B) by 

u{Y){k) := {Y,Adik)Ho) 

and note that 

u{Ad{h)Y){k) = {Ad{h)Y,Ad{k)Ho) = {Y,Ad{h~^k)Ho) = u(Y){h-^k) . 
Thus a; is a i^T-intertwining operator. 
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Note that ( ■ , ■ ) is negative definite on t. Let Xi, . . . , Xdimq be an orthonormal basis of q 
sucli tliat Xi, . . . , Xp, p = dim b, is an ortlionormal basis of b. Denote by f2 = — Xj the 
corresponding positive definite Laplace operator on B. The action of Q on functions on B 
is-up to a sign-the same as that of the Casimir element of t. Thus ^1^2 (g) = (/i + 2p{, /i)id. 
For later reference we state this ClS db Lemma: 

Lemma 3.2. For fi E A+(-B) /e^ 

= - f^/i(X,)(^ + 2pt)(X,) = -J2 {f^W + 2/i(X,)p^(X,)) . 
j=i j=i 
Then n|i2(g) = a;(yu)id. 

For / G C°^(i3) denote by M(/) : -4 ^^(S) the multiplication operator g fg. 

Theorem 3.3. Let Yes. Then [Q, M{u{Y))] = 2no{Y). 

Proof. This is Theorem 2.3 in [5]. □ 

For /i e A+(i3) define $^ : ® Sc ^ by 

<l>^(V9®r) :=M(u;(F))y.. 

Observe that for /c G -ft', F G Sq and G L'j^{B) we have 

Lfc(M(a;(y)y;) = = M(a;(Ad(A;)r)) (L^y.) 

with Ad(A;)y G Sc and Lk(p G L^(i3). Hence $^ is i^'-equivariant and Im$^ is i^-invariant. 
Define a finite subset S{fi) C A+(i3) by 

Im<|.^^;, LliB). 

aeSitJ.) 

Denote by pr^ the orthogonal projection L'^iB) — )■ L'^^iB). 

Lemma 3.4. Assume that fi G A^{B). Let a G S{fi), F G 5c (^T^d '^ot G a^- Let 

UJ,,{Y) := pr, o M(a;(r))U2(e) : Lj(i3) ^ . (3.1) 

Then 

pr^ o 7r,,(y)U2(e) = - + 2r)a;^^(y) . (3.2) 

Proof. This is Corollary 2.6 in [5]. □ 
Theorem 3.5. Let p G A+(i3), a G ^(/i) and A = ra G a^. T/ien 

^a(^) ^ 2r-a;(ff) +a;(p) 

?7^(A) 2r + a;(o-) - a;(p) ' 

r/ie K-spectrum {r7^(A)}^gA+(e) and /ience J(A) is uniquely determined by /13.3\) and the 
normalization rio{\) = cp(A). 
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Proof. Applying J(A) to (13. 2 p from the left, using that J(A) commutes with pr^ and that 
J(A) o nx{Y) = n^_^{Y) o J(A) = -Tr^x{Y) o J(A), we obtain: 

As u}as{Y) is non-zero for generic A it can be canceled out. The last statement follows from 
the fact that 7T\ is irreducible for generic A, hence iterated application of (13.11) will in the 
end reach all iiT-types starting from the trivial i^-type. □ 

4. The case of the Grassmann Manifolds 

From now on K denotes the one of the fields M or C, or the skew field EI of quaternions. 
We let G = SL(n + For discussion about the determinant function on the space of 

quaternionic matrices see [Hill]. We will use that if x G M(?t,+1, K) then |detiRa;| = |detKa;|'^, 
where d = dimuK is equal to 1,2,4 for K = M, C,EI, respectively. Denote by Gp(K) the 
Grassmann manifold of fc-dimensional subspaces of K""*"^. Note that Gp(K) ~ G„4_i_p(K). 
So we can assume without loss of generality that 2p < n + 1. We set q = n + 1 — p. Hence 
q > p. 

In this section we show that Gp(]K) = G/P = K/L, where P is a maximal parabolic 
subgroup of G. Furthermore K/L is a symmetric space. Therefore the results from the 
previous sections apply to this case. We introduce the Cos^-transform for K/L and then 
show that it agrees with the intertwining operator J(A). In the next section we then apply 
the spectrum generating operator to determine the i^-spectrum. 

Define an invariant M-bilinear form on g by 

77-1-1 

{X,Y) := ^^Re(Tr(Xr)). 
pq 

Denote hj z z the conjugation in K and for x = {x^^) G M(77 -|- 1, K) let x := (x^^) and 
X* := ix^u) = X* ■ The homomorphism 6 : G ^ G, x ^ (^"^)* is a Cartan involution on G. 
The corresponding Cartan involution on g is 6{X) = —X*. We have 

K = SU(77 + 1,K) := {x G G I X* = x-^} 

t = {X e M{n + 1, K) I X* = -X and Tr(X) = } 

s = {X e M{n + 1, K) I X* = X and Tr(X) = } 

In the notation in [13j we have G = SL{n + 1,M) and K = SO{n + 1) for K = M, 
G = SL(77 + 1, C) and K = SU(77 + 1) for K = C, and G = SU*(2(77 + 1)) and K = Sp(77 + 1) 
for K = M. 

For /c G N denote by 1^ the k x k identity matrix. We write matrices as blocks 

X G Mpxp(K) , Y G Mpx,(K) , W G M,xp(K) and Z G M,^,{K) . 
Let 

° V -;^tV ' 



X Y\ 

\w zj 
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and define o := RHo, and m = {X G 3g(a) | (X, Ho) = 0}. Then ^(m) fl 5 = a. We have 
A = {a,— a} where a{Ho) = 1. As {Ho,Ho) = 1 it follows that (•, •) agrees with the 
invariant form from last section. 



Let A+ = {a}. Then 

{M(x,y)^(^- «) 



X e M(p, K) 
Y e M{q, K) 



and TrX + TrY = 



n = 



s{Ql{p,K) X Ql{q,K)) , 
N{X) 



MA 



N 



m{a, b) :- 
S(GL(p,K) X GL(g 



a 
b 



X e M{p X g, K) I ~ M(p X q, K) 
and deta det6 = 1 



a e GL{p, K) 
b e GL(g, K) 



n{X) 



P 

L 



a X 
b 



X e M(p X q,K) 

a e GL(p,K) 
b e GL(g,K) ' 



detadetft = 1, X e M(p x q, K) 



p{a,b;X) := 
S(U(p,K) X U(g,K)). 

Let ei, . . . , Cn+i be the standard basis for K"""*""^ and let bo = Kci ® . . . © Ke^ e Grp(K). 
Then Grp(K) = K ■ bo ^ K/L = G/P. Thus we can introducce the notation B — Grp(K), 
which agrees with the one from the previous sections. Note that L — S(0(p) x 0{q)) for 
K = R,L = S(U(p) X U(g)) for K = C, and L = Sp(p) x Sp{q) for K = H. In particular B 
is simply connected for IK = C and H. 

Applying 9 to n, we see that 



n 



N{Y) 



Opp Opg 



and 



N = <^ n{Y) 



Ip Opq 

Y L 



FeM,xp(K)| ~M(?xp,K) 
Y e M{q X p, K) 



The action of M^l on n, respectively n, is given by 

m(a, b)N{X)m{a-^X^) = N(aXb-^) and m(a, b)N{Y)m{a-\ b'^] 



N{bYa-^) . 



Lemma 4.1. NP = 



A B 
C D 



e G 



A e GL(p, K) \ . 



Proof. A simple calculation shows that 



n{Y)m{a,b)n{X)^{^^^ FaX^ ft) ' 



(4.1) 
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proving one of the inclusions. On the other hand, let x = be such that A is 

invertible. Define a = A, X = a'^B, Y = Ca'^ and 6 = D - Ca'^B. Then m{a, h) e MA 
and n(Y)m{a^h)n{X) = x. □ 

Let b G Grp(K) and view 6 as a rfp-dimensional real vector space. Fix a convex subset 
E G b containing the zero vector, such that the volume of E is one. For c G Grp(K) let 
Pc : IK" — > c denote the orthogonal projection onto c. Define 

|Cos(6,c)| := Vo1m(Pc(^))'/' (4.2) 

The definition is independent of E as we will see in a moment. Recall, ii x : b ^ b is linear 
then 

Vohix{E)) = \detRix)\ = \dct{x)f. (4.3) 

As the elements in K acts as orthogonal transformations it follows that |Cos(A; ■ b,c)\ = 
|Cos(6, k^^c)\. In particular, if b = k ■ bo and c = h ■ bo with k,h E K, then 

|Cos(6,c)| - \Cos{h-'k ■ bo,bo)\ = Yoh{P,^{h-'kE)y/''. (4.4) 

As a motivation for the definition assume that p — 1 and K = R. If 6 = Mx, c — My G 
Gri(M), then 

|Cos(6,c)| = i^ = |cos(<(x,y))| 

where <{x,y) denotes the angle between x and y. 

For reasons that will become transparent in a moment, we identify a£ with C by 

A I-)- MHo) with inverse z i-^ z a . (4.5) 

pq ^ ^ n+1 ^ ^ 

We have dimK n = dpq. Hence 

dpq din +1) ^ 

'^2 2 

Then p is an integer if and only if K = C, H or K = M and n is odd. For a complex number 
A, we write from now on ap{x)'^ instead of «p(a;)'^"+T". 

Theorem 4.2. Fix a convex subset E d bo containing zero such that Wol^{E) — 1. For 
X E G and X E C we have 

ap{xy = \Voh{P,MEm^^' = |Cos(x • bo,bo)\\ (4.6) 

In particular, if b — k ■ bg and c — h-bgEB, then 

a{h-^k)^ =\Cos{b,c)\\ (4.7) 
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Proof. For z 



Thus 



Thus, by (US), 



G W we have 

A B 
C D 



Az 
Cz 



bo 



A B 
C D 



Cos 



A B 
C D 



• ho, ho 



Az 




IdetM^I 



\/d 



(4i 



Let n{Y)m{a,h)n{X) G NP. Write m{a,h) = moexp(a;i7o) with rrio G M and x G 
Then 



I 1 . |A / \ xdpq\ 

IdetiRal = exp A 

n + lj 



exp{xHo) 



n + 1 



ap(exp(xifo)) 



dX 



Thus 



IdetiRfa)!^/'^ = Idetal 



(4.9) 
□ 



ap{n{Y)m{a,h)n{X))^ 
and fl4.6p follows now from fl4.8p . 

This shows in particular that the definition of Cos is independent of the convex set E 

Lemma 4.3. Let k E K. Then ap{k) = ap{k~^). 

Proof. Write k = nman. Then, as 6{k) = k and 6{a)~^ = a, we have k^^ = 9{k)^^ 
9{n)~^9{m)~^a9{n)~^ and the claim follows. 

Corollary 4.4. Leth,ceB. Then |Cos(6,c)|^ = |Cos(c,6)|^. 

Proof. Write h = k ■ ho and c = h ■ ho- By (14. 7p and Lemma [4.31 

|Cos(6,c)|^ = a{h~^k)^ = a{k-^hY = \Cos{c,h)\\ 
which proves the Corollary. 

The Cos'^-transform C'^ is defined for Re A > p by 

C\f){c):= [ \Cos\h,c)\^~ff{c)dc= [ a{k^^h)^-Pf{h-ho)dk c = k ■ ho 



□ 



□ 



(4.10) 



where we have used Theorem 14.21 Thus, C'^ is nothing but the standard intertwining 
operator J(A). We would like to point out, that the proof of the intertwining property of 
J(A) shows that is an intertwining operator without the reference to the general theory. 
But the general theory described in Section [2] gives us the following theorem for free and 
allows us to apply the results from [5j described in Section O 



Theorem 4.5. The following holds: 



[1) The Cos^ -transform extends to a meromorphic family of intertwining operators : 
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(2) Let cp(A) := Then cp : C — > C is meromorphic and 

C-^ oC^^C^o C-^ = cp(-A)cp(A)id . 

(3) If X & iR; then -^^^C^ is unitary. 

5. The /T-spectrum of the Cos^-Transform 

In this section we determine the i^-spectrum of J(A) for the Grassmann manifolds Gp(K) = 
SU(n + l,K)/S(U(p,K)xU(g,K)) ~ SL(n+l,K)/P. This then determines the fC-spectrum 
of the cos'^ transform. As before we always assume that p < q and p + q = n + 1. We keep 
the notation from the previous sections. 
The following is well known. 

Lemma 5.1. The space B is a symmetric space and the corresponding involution is 

^^""^ = (o -I J ft -i„) = i-c ~d) ^ = D 

and K'^ — L. 



The derived involution is given by the same formula. Let I denote the Lie algebra of L. 
Then t = I © q where 



Q{X) 



-X* Ogg 



X e M(p X q, K) 



Let -B^y/f^ = {5iv5jtj)i,j denote the matrix in M(r x s, K) with all entries equal to but 
the {v, //)-th which is equal to 1. For t = (ii, . . . , tpf e R** let 

Y{t) = Q(X(t)) e q 

V 

diag(t) = h^3,3 e M(p X p, K) . 

i=i 

Then b = {Y{t) \ t G MP} ~ is a maximal abelian subspacc of q. 

To describe the set Kl we note first that if K = R, then B is not simply connected. In 
all other cases B is simply connected. Define ej{Y{t)) := tj. We will identify the element 
A = Y2^=i with the corresponding vector A = (Ai, . . . , \p). 

Lemma 5.2. We have 

A{ = {±ej ± ej (1 < i 7^ J < p , ± independent ), icj (1 < i < p) , ±2ej (1 < i < p) } 

with multiplicities respectively d (and not there in case p = I), d{q — p) (and not there in 
case p — q) and d — 1 ( and absent if d— 1). 
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Proof. This follows from [13]: the table on page 518, the description of the simple root 
systems on page 462 ff. and the Satake diagrams on page 532-533. □ 

Lemma 5.3. Let the notation be as above. 

(1) IfK = R andp = q, then 

p 

J\.'^{B) = {/i = ^j^j I ^ 2No and mi > . . . > > \mp\ } . 
i=i 

In all other cases, 

p 

A^(i3) = {/U = ^i^i I £ 2No and mi > . . . > mp > } . 

i=i 

(2) Pt = i:U[p-d{j-l)-lh- 

p 

(3) Let fi e A+iB). Then = J"^ Yl H + '^'^'^^ ~ '^^^ ^0 ' 

+ j=i 

Proof. (1) is well known and (2) follows by a simple calculation using the form of and 
the multiplicities. For (3) let Xi, . . . ,Xp G b be so that ej{Xk) = iSjk- Then —{Xk,Xk) = 
2{n + l)/pq. Thus the = ^ypq/[2{n + 1)] form an orthonormal basis for b and 
— ej(Xfe)^ = pq/[2{n + l)]Sjk- The claim now follows from Lemma [3.21 □ 

Lemma 5.4. Let fi = {mi, . . . ,mp) G A'^{B). Then 

S{fi) = {fi±2ej\j = l,...,p}nA+{B). 

These representations occur with multiplicity one. 

Proof. A simple matrix calculation shows that the weights of I) in 5c are ±(ei + ej) for 
i,j = 1, . . . ,p. Next we note that /z ± (e^ + ej), j ^ i, is not in A^{B) as the i^^ and j*^ 
entries are not even. It follows now from [T7] and the fact that each submodule in ® Sc 
that occur in L'^{B) is spherical that cr G S{fi) if and only if there exists j = 1, . . . ,p and 
w G Wf such that ± 2ej) is dominant and a = w{fi±2ej). Assume that /i + 2ej is not 
dominant. Then mj^i < mj + 2. Hence mj = mj^i. Applying the transposition (j — 1, j) 
shows that /i + 2ej is PV{-conjugate to /i + 2ej_i. Repeating as long as necessary shows that 
there exists a. k, 1 < k < j such that fi + 2ek is dominant and PVt-conjugate to /i + 2ej. The 
case fi — 2tj not dominant is handled in similar way. □ 



Lemma 5.5. Let jj, = (mi, . . . ,mp) G A+(i3) and A G C ~ aj. Then 

^M+2e, (A) ^ A - mj - p + - 1) 
ViiW \ + mj + p - d{j - 1) ' 



{5.1] 
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Proof. By Lemma [5.31 we get 

uj{fi + 2ej) - uj{fi) = {mj +p- d{j - 1)) . 

The claim now follows from fl3.3p and our identification C ^ a^, z i— z^^a. □ 
Introduce the Siegel or Gindikin F-function Fp ^ : — )• C by 

rM(A) = nr(A,-^(j-i)). (5.2) 

Then T{p, d; A) is meromorphic with singularities on the union of the hyperplanes 

nid,j,n) = {A G I A, = -n + ^{j - 1)} , j = 1, . . . n G No . 

In the following, if A G C, we will also indicate by A the vector (A, . . . , A) G C^. It will be 
clear from the context if we are using A as a complex number or as a vector. 

Theorem 5.6. Let A G C. Then there exists a meromorphic function G{X) such that for 
all fi G A+(i3) we have 

,,(A) = (-l)l-l/^G(A)^^4feT^^r^ ' (5.3) 

where \fi\ = 'Yl^j=i^j- 

Proof. Denote the right hand side of (15.31) by ^"^^(A). Using V{z + 1) = zV{z), we get 

C^/.+2.,(A) = (T^(A) 



\ — nij — p + d{j — 1) 



X + rrij + p - d{j - 1) 

Now Lemma [5.51 implies that there exists a meromorphic function G{X) with the requires 
property. □ 

The function G{X) will be explicitly computed in (15. 8p below, by comparing the formula 
for ?7o(A) = cp(A). For this, we note the following result which can be found in Theorem 
8.1.1. 

Lemma 5.7 (Selberg's integral). Let 71,72, a G C with Re7i, Re72, Rea > 0. Then 

['■■■ffl^r'(^-^^y''' n \u-t,rdt,...dt,= 

Jo Jo l<j<7<P 



n 



<j<p 

T{aj + l)F(a(j - 1) + 7i)r(a(j - 1) + 72) 



.^^ F(a + l)F(a(p + j - 2) + 71 + 72) 
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To complete the calculation we need to evaluate ap(exp(F(t))). We have 

(diag(costi,..., costp) — diag(sinti, . . . , sintp)\ 
Vp (5.4) 

diag(sinti, . . . , sintp) diag(cosi(:i, costp) J 

where the matrix Ig_p = I„+i_2p does not occur in case p = {n + l)/2. 
For the following see also [21], Lemma 3.2: 

p 

Lemma 5.8. Let teW andXe C. Then ap(expr(t))^ = JJ | cos(tj)|^. 

i=i 

Proof. This follows from ([S3D, dSSD and (jH]). □ 
Theorem 5.9. Let cp(A) = J^^ ap{ky-P dk = r]o{X) = J(A)1. Then 

rp,,(d(n + l)/2)rp,,(|(A-p + rfp)) 

r,Adp/2) rp,4i(A + p)) • ^^-^^ 

Proof. In the following c will denote a positive constant that might change from one line 
to another. Its value will then be determined at the end. Let 7i = (A — p + d)/2, 72 = 
d{q - p + l)/2, Q = [0, and = [0,n]P. Using that sin(2M) = 2 cos(m) sin(M) and 
I sin(u — v) sin(u + f)| = | cos^(u) — cos^(i;)|, we get by the substitution uj = cos^(tj): 

r ^ 

cp(A) = c 



f f[ I costjf ^1-^ sin(tj)2T2-i Yl I cos^(ii) - cos'^{tj)\'^ dti ...dtp 
/ Yl I costjf ^1-^(1 - cos^tj)^^-! gjj^(^^.) JJ I cos2(ti) - cos'^ {tj)\'^dti ...dtp 

r ^ 

■^Q 7 = 1 i<7 



^ T{l{X-p + dj)) 

^-^r(i(A-p + dp-d(j-i))) 



,=1 r(i(A + p-rf(j-i))) 

In the last equation we used that d{q + j) = d{n + 1 — p + j) = 2p — dp + dj and that for a 
complex number z 

f[T(^iz + dj))=flT(hz + dp-dij-l))) . (5.6) 
i=i ^ ^ i=i ^ ^ 

Now evaluate c by using that cp(p) = 1 and recalling that p = (i(n + l)/2. □ 
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Theorem 5.10. The K-spectrum the Cos -transform is given by: 

^ 1^1/2 r,^,{d{n+l)/2) r,,, (|(A - p + dp)) Tp,, (i(-A + p + fi)) 

"^'^ ^ ^ ^ r,Adp/2) r,,,(i(-A + p))r,,4i(A + p + p)) ' ^''^^ 

Proof. By taking p = in (15. 3p and using rjQ = cp, we get 

G(A) = ^^#i^.p(A). (5.8) 

The claim then follows from (15. 3p and (15. 5p . □ 

Remark 5.11. In Section [2] we referred to the result of Vogan and Wallach on the mero- 
morphic continuation of the intertwining operator J(A). This result is not needed for the 
computation of ri^{X). Indeed, it is enough to know that J(A) is holomorphic on Oc(ap). 
Then Theorem 15.101 implies the meromorphic continuation of J(A) = at least at the 
level of i^-finite functions, cf. [T6l. 



Remark 5.12. One can give a more general definition of the Cos^-transform. Note first 
that the definition of |Cos(6, c)|'^ in (14. 2 p is valid for all b G Grp/(K), p' > p. For p' < p 
one uses the symmetry relation |Cos(6, c)|'^ = |Cos(6, c)|'^. For p < p' define the Radon 
transform Rp,^p : C'^(Grp/) ^ C°°(Grp) by 

Rp',p{f){uj) := j f{ri)dap,^^{ri) 

J u)Cr] 

where cxp/ is a well defined probability measure on the set {rj \ u C rj}. Then Rp'^p is a 
i^-intertwining operator whose i^-spectrum is independent of A. Furthermore, defining 
by (I4.10p . one shows that, with the obvious notation, 

^p',p = o Rp, p . 

The determination of the ii'-spectrum of the more general Cos^-transform is then reduced 
to the determination of the A-independent ii"-spectrum of Rp\p and the determination of 
the A-dependent i^'-spectrum of C^. We note that obviously Rp'^p is zero on all -ft^-types 
p G A~^{Bp') \ A+(i3p) n A+(i3p/), or equivalently, have a nonzero entry nij with p < j < p' . 
As our purpose is to relate the Cos'^-transform with the standard intertwining operators, 
we do not go further into this interesting topic, but refer to [31] and the references therein 
for further discussion. 

6. The Sin^-TRANSFORM 

In this section we introduce the Sin'^-transform and determine its /T-spectrum. We assume 
that p = q. This in particular implies that n + 1 is even and p = If a; G Grp(]K) then 
denotes the orthogonal complement of u. Define the Sin''*-transform 5^ : — > C°°{B) 
by 

S\f){uj) :=C\f){u^). 
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This transform has been studied by B. Rubin for more general p, see [28l [29] and refer- 
ences therein, and by G. Zhang [31j. In particular G. Zhang shows, using the realization 
of the principal series representations, that the Sin'^-transform is a Knapp-Stein intertwin- 
ing operator and uses this fact to determine the complementary series for vr^. 

As p={n+ l)/2 we have Ho = (^^^^ _ ? j ^ . Let 
Then wHoUJ-^ = Ad{w)Ho = -1. It follows that Ad(w)iV = A^. 

Lemma 6.1. For f e define A{X)f{k ■ bo) := J{X){f){kwbo) = R{w) J {X) f {k ■ bo) , 

where R{w) denotes the right translation by w. Then A{X) : C°°{B) C°°{B). Then 
X I— 7- A{X) is a meromorphic family of intertwining operators: A{X) o nxlx) = 7i^\{x)A{X) . 

Proof. We have w~^Lw = L as Ad(w) normalizes a. Thus kmw ■ bo = kw ■ fo^ if m G L and 
k E K. Thus A{X) is well defined. The intertwining property follows from Theorem 12.31 
and the fact that w~^Nw = N and Ad(i(7)|a = —1. □ 

Theorem 6.2. If X e a£(ap), then A{X) = S^. 

Proof. We have by definition 

A{X)f{k-bo)= f \Cos{h-bo,kwbo)\^-Pfih)dh. (6.1) 

The claim follows if we can show that (k-bo)^ = kw-bo- As SU(n + l,K) acts by orthogonal 
transformations it is enough to show that b-^ = w ■ bo- But thit follows from 

w{xi, . . . , Xp, 0, . . . , 0)* = (0, . . . , 0, xi, . . . , Xp)* . □ 

As for the Cos'^-transform, it follows that for each // G A+(i3) there exists a meromorphic 
function z/^(A) such that 

As G L it follows that R{w) defines a unitary i^-intertwining operator of order two 
on each of the if- types L'j^{B). Thus R{w)\l2(^q^ = r(yu)idL2(g) where r(/i) is a constant 
independent of A, r(/i)^ = ±1 and z^^(A) = r{^)rj^{X). 

Lemma 6.3. Let fi G A+(i3). Then r{^) = (-l)l'^l/2. 

Proof. Let G be as in Section [2l Then 7r^(w)e^ = r{n)e^. Let be a highest weight 
vector such that (f^,e^)v^ = 1. Then 

r(/x) = {v,,7,,{w)e,) = in,iw-')v„e,) = e'^^"^^ {v„e,) = (-1)1^1/^ □ 
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Theorem 6.4. The K -spectrum of the Sin^ -transform is given by 

^pAp) rp,4|)rp,4|(-A + /i + p)) 



rp,d(p/2) Tp,, (1(-A + p)) Tp,, (i(A + P + p)) 
wzt/i p = dp. 

Proof. This foUows from Theorem 15.101 and Lemma 16.31 □ 
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